1. Field of the Invention
The present invention relates generally to systems and methods for splicing two compressed bitstreams together to form a single compressed bit stream. In particular, the present invention relates to a system and a method for video bitstream splicing that is frame accurate. Still more particularly, the present invention relates to a system and method for seamless splicing of MPEG-2 video bitstreams.
2. Description of the Related Art
Digital video compression is a process that removes redundancy in the digital video pictures such that the resulting representation of the signal contains much smaller number of bits than that in the original uncompressed pictures. The redundancy in the digital video sequences, consist of a sequence of digital pictures played out in a time continuous manner, is reflected in the form of spatial redundancy in a picture and temporal redundancy between pictures. MPEG-2 compression takes advantage of these redundancies by efficient coding of the spatial digital image content and temporal motion content. Algorithms for MPEG-2 video compression are well known in the art.
Digital stream insertion (also called digital program insertion (DPI), digital spot insertion, etc.) is a process that replaces part of a digital compressed bitstream by another compressed bitstream, which may have been encoded off-line in a different location or at a different time. This process is illustrated via FIG. 1. In the figure, part of bitstream 1 is replaced by bitstream 2. In real applications, bitstream 1 may be a real-time feed from the distribution network, and bitstream 2 may be a section of advertisement that is to be inserted into the network feed. As a result of the insertion, the resulting bitstream has the advertisement inserted into the network bitstream feed. Since this is the main application of DPI, we may refer in this application to bitstream 1 as the live stream, and bitstream 2 the stored stream.
The underlying technique for DPI is bitstream splicing (also known as bitstream concatenation), where a transition is made from an old stream to a new stream. The transition is called splice point. The splicing process in its most general form could be between a first and a second stream that are continuously playing and the transition is from the middle of the first stream to the middle of the second stream. However, in the context of DPI, two special cases are of interest. Each insertion involves two splice points: an exit point and an entry point. Here, we define the exit point as the transition from the middle of the first (live) stream to the beginning of a second (stored) stream. We define the entry point as the transition from the end of the second (stored) stream to the middle of the first (live) stream. Both of these points are splice points, and are illustrated in FIG. 1.
One prior art method for performing splicing is the use of analog splicing equipment. In this case, the two signals to be switched are time continuous pictures. The splicing equipment, at the time of splicing, turns off one signal at the picture frame boundary and turns on the other signal, resulting in a scene change to the viewer. The two signals are assumed to be frame synchronized, the time for the splicing is well defined. However, to splice two digitally compressed bitstreams, the situation is much more complex. This is due to the nature of the motion compensation and variable length encoding of digital video pictures. Specifically, compressed or encoded digital pictures do not necessarily have the same number of bits; in addition, the digital picture content is reconstructed not from a single encoded digital picture, but from several of them via motion compensation. More specifically and as show in FIGS. 2A and 2B, the bitstreams are compose of a number of frames or pictures. The MPEG standard defined three types of pictures: intra, predicted and bi-directional. Intra pictures or I-pictures are coded using only information present in the picture itself. Predicted pictures or P-pictures are coded with respect to the nearest previous I- or P-picture as show in FIG. 2A. Bi-directional pictures or B-pictures are coded with reference to the two most recent sent I/P-pictures as illustrated by FIG. 2B.
The basic requirement for any digital bitstream splicing system or method is that the splicing must not introduce any visible artifacts into the decoded video of the resulting bitstream. There have been some prior art method for digital bitstream splicing, however, they are not able to provide frame accurate splicing. These prior art method resort to imposing several constraints to the formats of the streams being spliced, thus not providing seamless and frame-precise splicing. For example, some prior art methods allow splicing only immediately before an I-frame. This is problematic because there could be as many as 30 frames between I-frames. Other methods require that the bit rates of the spliced streams be the same and constant. Yet other prior art methods require that the stored stream start with an I-picture and/or the live stream start with an I-picture right after the stored stream has ended. Thus, such prior art methods do not provide for seamless and frame-precise splicing where splicing is allowed to happen at any picture location (this is what frame-precise or accurate splicing means).
A particular problem with the slicing methods of the prior art is that they require both streams to have the same constant bit rate. Splicing of bitstreams may occur in either constant bit rate (CBR) environment or variable bit rate (VBR) environment. In the CBR environment, the live stream has a bit rate, R1, that is constant throughout the transmission. In order to splice the stored stream with bit rate, R2, the two bit rates must be identical, R1=R2. For example, assuming the channel transmits at rate R1 before and after splicing, if R1 greater than R2, then the decoder buffer will overflow, shortly after the exit point, causing corruption in the decoded pictures, and if R1 less than R2, the decoder buffer will under flow, shortly after the entry point, which causes the decoder to freeze displayed pictures. This rate mismatch can be solved by either stuffing of the stored stream if R1 greater than R2, or by rate reduction if R1 less than R2. Thus, the prior art has not provided a splicing system and method able to handle streams with different rates in real time.
Another more generalized look at the same problem described above is the buffer compliance problem (i.e., the problem of matching coded picture bit usage to that of the data delivery rate of the transmission channel). This problem is what is called the rate control problem. The MPEG-2 encoding process must be under a rate control algorithm. Specifically, the encoder rate control must ensure that the decoder buffer cannot under flow or overflow. If no splicing is performed in the bitstream, this responsibility is taken entirely by the encoder. However, when splicing is performed, two bitstreams are involved, each is possibly encoded at a different time by a different encoder. In this case, the decoder""s buffer trajectory just before splicing is defined by the rate control algorithm running on the live stream encoder. Starting from the splicing point, the new stream encoding rate control takes over, and this is where the buffer compliance of the decoder may be violated. To see this, consider the following example shown in FIG. 3. The figure describes the decoder buffer behavior during the splicing. In the example shown, assuming two CBR bitstreams are spliced together without change, and assuming that the first picture of the new stream replaces the B picture right after the last I1-picture in the old stream is removed from the decoder buffer. Since the first stream encoder knows that the next picture in the first stream is a B2-picture, it will thus assume that the decoder buffer will not under flow given that B2 has fewer bits. However, unknown to the second stream encoder, this is where splicing occurs. The next picture to be removed from the decoder buffer is actually the I2-picture from the new stream, which has many more bits than the B2-picture in the first stream. This will causes the decoder buffer to under flow as shown in the FIG. 3. Thus, there is a need for a digital frame splicer that eliminates the under flow or overflow of the decoder buffer. Again, the problem described above is not limited to CBR to CBR splicing, but exist also in more general variable rate bitstream splicing.
A further problem in splicing two digitally encoded streams is resolving timing differences between the two streams. The splicing process causes two different MPEG-2 bitstreams, each encoded in a different location or at a different time, to be joined into a single bitstream. Therefore, the first and the second streams may contain independent timing information, which was generated by its respective encoder. The splicing process may cause a discontinuity in the time base. The spliced stream must be decoded in real-time by the decoder. The decoder maintains a local clock which runs phase locked to the time base embedded in the input bitstream. Therefore, the discontinuity of the time base due to splicing may cause the decoder to temporarily lose synchronization. The prior art has attempted to solve this problem by inserting a PCR carrying packet as the first packet of the new stream and at the same time setting the discontinuity indicator in the adaptation field to 1. An MPEG-2 compliant decoder receives this indicator and takes proper action to ensure the time base discontinuity is handled correctly. This requirement on the PCR packet placement, however adds additional complexity and hardware, and is not a complete solution because some decoders do not recognize the indicator. Thus, there is a need for splicing two digital bitstreams without the above timing problems.
A final problem in splicing two digitally encoded streams is the dependence of frames before the splicing point on frames after the splicing point. Due to the inter-frame coding, a decoded picture requires the use of several coded pictures. For example, if a particular picture is coded as B picture, several other neighboring coded pictures must also be used to first obtain reference pictures, then the coded pictures. This presents a particular problem because after splicing those frames will not be part of the bitstream. If a B picture is the last picture to be displayed before the new stream is inserted, the decoder will use the next anchor picture in the new stream as the reference picture to decode the B picture, which will cause artifacts in the decoded pictures. Similar problem exists upon completion of the splicing. When the stored stream is completed, the splicer needs to switch back to the first or live stream. If the first picture after the switch back is not an intra picture, the decoded picture may not be correct. The prior art has attempted to solve this problem by requiring that the real-time encoder of the live stream forces anchor pictures at exit and entry points, closed GOP restriction, insertion of splice point syntaxes into the live stream. However, these restrictions limit the flexibility of the splicing process and are generally difficult to implement. Another alternative is to use real-time decoder followed by encoder to perform real-time re-encoding which can re-encode and insert anchor pictures at splicing points. The difficulty with this approach is that such real-time decoders and encoders are very expensive in terms of cost, computation, area of silicon and a variety of other factors.
Therefore, there is a need for a new system and a new method for performing seamless splicing of bitstreams at a frame accurate level.
The present invention overcomes the deficiencies and limitations of the prior art with a system and a method for seamless, frame-accurate bitstream splicing. The present invention provides arbitrary splicing of bitstreams in real-time. The system for performing frame accurate bitstream splicing comprises a first pre-buffer, a second pre-buffer, a seamless splicer, and a post-buffer. The system also includes a time stamp extractor, a time stamp adjuster and a time stamp replacer to adjust the time base to match the spliced stream output by the seamless splicer. The first pre-buffer is coupled to receive a first bitstream, and the second pre-buffer is coupled to receive a second bitstream. The output of each of the first and second pre-buffers is coupled to a respective input of the seamless splicer. The seamless splicer receives the two streams via the first and second pre-buffers and produces a single spliced bitstream at its output in response to the cue tone signal. The seamless splicer provides the first bitstream, and then re-encodes portions of the first and second bit streams proximate the splicing points (both the exit point and the entry point). The seamless splicer also performs rate conversion on the second stream as necessary so that the spliced stream has decoder buffer compliance. It should be understood that throughout this patent application the terms xe2x80x9cre-codingxe2x80x9d and xe2x80x9crate conversionxe2x80x9d have been used interchangably. The output of the seamless splicer is coupled by the post-buffer to the time stamp replacer. The use of the first and second pre-buffers and the post-buffer is particularly advantageous because it allows effective real-time re-encoding to be performed without the need for the complex and expensive hardware required for real time encoders and decoders.
The present invention also includes a method for performing bitstream splicing comprising the steps of: determining a splicing point switching between a first bitstream and a second bitstream, determining whether the second bitstream has the same bit rate as the available transmission bandwidth, converting the rate of the second bitstream if it is not the same as the bit rate of the transmission bandwidth, regardless of whether it is CBR or VBR, and re-encoding picture proximate the splicing point.
The system and method of the present invention is particularly advantageous because it imposes minimal constraints on the bitstream to be spliced, and can be used to perform unrestricted frame-precise splicing regardless of the original coded picture type. Furthermore, it can perform the unrestricted frame-precise splicing in real-time. In particular, the present invention provides a unique and novel rate matching of the two input bitstreams. The present invention also ensures the buffer compliance for the decoder with rate control. Finally, the present invention also corrects the differences between the two time bases of the streams being spliced through the use of a time stamp adjuster that corrects the PCR and PTS/DTS of the new spliced stream.